Skip to content

test: convert silent t.Skip to t.Fatal in baseline-scan checks#8

Closed
omercnet wants to merge 1 commit into
mainfrom
coverage/pr6-skip-to-fatal
Closed

test: convert silent t.Skip to t.Fatal in baseline-scan checks#8
omercnet wants to merge 1 commit into
mainfrom
coverage/pr6-skip-to-fatal

Conversation

@omercnet
Copy link
Copy Markdown

@omercnet omercnet commented Apr 29, 2026

Summary

  • Converts 4 silent `t.Skip` calls into `t.Fatal` in e2e tests where the skip was masking real regressions.
  • Adds a reason string to one legitimate skip in `integration/singlearch` for log readability.
  • No production code changed.

Why

Three sites in `test/e2e/golang` and one in `test/e2e/frontend` were silently skipping when their baseline assertion failed. This masks real regressions:

`test/e2e/golang/golang_test.go` (`TestGoBaselineImages`, `TestMultiBinaryImages`, `TestDistrolessImages`)

Each test selects a baseline image specifically because it carries known CVEs. If the baseline scan returns zero vulnerabilities, the test should FAIL loudly — it indicates one of:

  • A scanner regression
  • A stale Trivy DB
  • An upstream image rebuild that resolved the CVEs (test data needs updating)
  • A network failure during DB download

Silently skipping made the test green when in fact the test was unable to assert anything.

`test/e2e/frontend/frontend_test.go` (`TestMultiplatformFrontend`)

When every platform's report generation fails, the test was skipping with a hand-wavy "possible disk space or network issue" message. Per-platform failures are already surfaced via `t.Logf` within the loop; if the cumulative result is zero successful reports the failure is real, not flake.

`integration/singlearch/patch_test.go` (legitimate skip, just needs a message)

The skip for local-name images on non-docker buildkit is correct — local-only images can only be tested with docker:// daemon. But the previous `t.Skip()` had no message, making gotestsum output unreadable. Adds a `Skipf` with image ref + current `COPA_BUILDKIT_ADDR`.

Risk

Could turn currently-green CI red if the underlying assumption is wrong (e.g., a baseline image was rebuilt upstream and now has 0 CVEs legitimately). If that happens, the right response is to update the baseline test data, not to re-silence the failure.

Test plan

  • `go vet ./test/e2e/golang/... ./test/e2e/frontend/... ./integration/singlearch/...` clean.
  • The next CI run on this branch will surface any baseline that's already in the failure mode.

Summary by cubic

Convert silent t.Skip to t.Fatal in baseline-scan checks so real regressions fail fast in CI. Also add a message to one legitimate skip for clearer logs. No production code changes.

  • Bug Fixes
    • test/e2e/golang/golang_test.go: Fail with t.Fatalf when a baseline scan returns 0 vulns; includes image context.
    • test/e2e/frontend/frontend_test.go: Fail when zero reports are generated across all platforms.
    • integration/singlearch/patch_test.go: Add t.Skipf message for local-only images on non-docker:// buildkit.

Written for commit 66b989e. Summary will update on new commits. Review in cubic

Three sites in test/e2e/golang and one in test/e2e/frontend were silently
skipping tests when their baseline assertion failed. This masks real
regressions:

- test/e2e/golang/golang_test.go (TestGoBaselineImages,
  TestMultiBinaryImages, TestDistrolessImages): each test selects a
  baseline image specifically because it carries known CVEs. If the
  baseline scan returns zero vulnerabilities, the test should FAIL
  loudly — it indicates a scanner regression, a stale Trivy DB, an
  upstream image rebuild that resolved CVEs, or a network failure
  during DB download. Silently skipping made the test green when in
  fact the test was unable to assert anything.

- test/e2e/frontend/frontend_test.go (TestMultiplatformFrontend): when
  *every* platform's report generation fails the test was skipping
  with a hand-wavy 'possible disk space or network issue' message.
  Per-platform failures are already surfaced via t.Logf within the
  loop; if the cumulative result is zero successful reports the
  failure is real, not flake.

Each Fatalf message includes enough context (image ref, mode) for the
on-call to triage without re-reading the test.

Also adds a reason string to the legitimate skip in
integration/singlearch/patch_test.go where local-name images are not
testable on non-docker buildkit drivers — this skip is correct, but
previously had no message which made gotestsum output unreadable.

No production code changes.
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

@omercnet
Copy link
Copy Markdown
Author

Superseded by upstream PR project-copacetic#1578 — closing this duplicate. CI never ran on this fork (no oracle-vm runners registered).

@omercnet omercnet closed this Apr 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant